SPBERT: an Efficient Pre-training BERT on SPARQL Queries for Question Answering over Knowledge Graphs

In this paper, we propose SPBERT, a transformer-based language model pre-trained on massive SPARQL query logs. By incorporating masked modeling objectives and the word structural objective, SPBERT can learn general-purpose representations in both natural language. We investigate how encoder-decoder architecture be adapted for Knowledge-based QA corpora. conduct exhaustive experiments two additional tasks, including Query Construction Answer Verbalization Generation. The experimental results show that obtain promising results, achieving state-of-the-art BLEU scores several of these tasks.